In another article, we touched a bit on generalization. What is the relationship between the generalization error and the training error? Generalization is the concept of the machine learning algorithm being able to produce good predictions on previously unseen inputs. The red line represents the training error. If the horizontal axis is the quantity of training examples or time, depending on how you like to think about it, then with time this training error gets smaller and smaller.

An agreed upon definition of machine learning is, a computer program is said to have learned when it's performance measure P at task T improves with experience E. Under the definition of Supervised Learning, we get this diagram. Here the experience would be the training data required to improve the algorithm. In practice we put this data into the Design Matrix. Design Matrix [dəˈzīn ˈmātriks]: term -- if a single input can be represented as a vector, putting all of the training examples, i.e the vectors, into 1 matrix makes the entire input aspects of the training data. This is not all of the experience. We still need the labels, if the examples are the inputs.